Queue push/pop RunBook - San BAU Team

Name

 queue_push -host <hostname> -operation <split|establish|failover|failback>
 queue_push -status
 queue_push -trans_log
 queue_push .help
 queue_pop -status
 queue_pop -accept
 queue_pop -trans_log
 queue_pop -help

Document Version

Version 1.0

08/11/06/Tim Robinson - Initial Document (Word Format)

Version 1.1

11/12/06/Tim Robinson - Document Conversion to POD format

Overview

Following initial infrastructure COB testing, it was noted that the SAN BAU team was responsible for a noteable delay in the time taken to failover applications. On inspection, it was agreed that this was attributed to a number of factors, one of which being the effective use of resources within the BAU team.

E-mail was found to be an unsuitable medium when queuing requests, as multiple administrators are working concurrently, which resulted in more than 1 administrator attempting to service the same request.

In answer to this, I have implemented a simple software queuing system, which can be accessed concurrently. This queuing system should contain a time-stamping mechanism to ensure that any SRDF operations performed within the team are recorded.

In addition to this, the queuing system should also record when the request has been satisfied, together with the administrator that has processed the request.

Process Description

The queuing mechanism should consist of 4 separate components:

  1. Queue structure
    2. This will store all current SRDF operations to be processed by the team.

  2. Queue_push operations
    3. This will be a script that can add work into the queue, consisting of a hostname and an SRDF operation. This should be used by the BAU COB coordinator to queue work for administrators to process.

  3. Queue_pop operations
    4. This will be a script that can allocate work from the queue. Multiple administrators will run this code, to allocate them a specific piece of work to do.

  4. Transaction Log
    5. This will be written to when an administrator is allocated a piece of work, and will contain the following fields :-

    Hostname

    Operation type (split/establish/failover/failback)

    Time stamp of queue entry

    Time stamp of allocation

Additional Notes

The queue_push program should not access the centralized queue when the queue_pop program is allocating work as this could lead to a concurrency violation (i.e. if both programs take a temporary copy of the queue for processing).

In the same way, the queue_pop program should not access the centralized queue when the queue_push program is being used to add information into the queue as this could also lead to a concurrency violation (see above).

This will mean that a lock_file will need to be used to prevent the above boundary condition from happening.

Author

Tim Robinson (2006)

See Also